System, Method, and Graphical User Interface for Controlling an Application Executing on a Server

ABSTRACT

A system, method, and graphical user interface for playing games and/or executing applications on a tablet-based client. One embodiment of a graphical user interface (GUI) for playing a video game on a tablet-based client device comprises: a virtual controller rendered on a display of the tablet computer, the virtual controller substantially mimicking the control provided by a thumb stick of a physical game controller and providing omnidirectional, free-form movement in a synchronous direction in which a user moves a finger on the display of the tablet-based client.

CLAIM OF PRIORITY

This application is a continuation of U.S. patent application Ser. No.16/660,762, filed on Oct. 22, 2019, and entitled, “System, Method, andGraphical User Interface for Controlling an Application on a Server,”which is a continuation of U.S. patent application Ser. No. 16/236,230,filed on Dec. 28, 2018 (since issued as U.S. Pat. No. 10,449,450 on Oct.22, 2019), and entitled, “System, Method, and Graphical User Interfacefor Controlling an Application on a Server,” which is a continuation ofU.S. patent application Ser. No. 15/797,589, filed on Oct. 30, 2017(since issued as U.S. Pat. No. 10,166,471 on Jan. 1, 2019), andentitled, “System, Method, and Graphical User Interface for Controllingan Application on a Tablet,” which is a continuation of U.S. patentapplication Ser. No. 13/797,039, filed on Mar. 12, 2013 (since issued asU.S. Pat. No. 9,868,062 on Jan. 16, 2018), and entitled, “System,Method, and Graphical User Interface for Controlling an Application on aTablet,” which claims priority from U.S. Provisional Patent ApplicationNo. 61/610,278, filed on Mar. 13, 2012, and entitled, “System, Method,and Graphical User Interface for Controlling an Application on aTablet,” each of which is incorporated herein by reference in itsentirety.

CROSS REFERENCE TO RELATED APPLICATION

This application is related to U.S. patent application Ser. No.12/538,077, filed Aug. 7, 2009, and entitled, “SYSTEM AND METHOD FORACCELERATED MACHINE SWITCHING,” which claims priority to U.S.Provisional Application Ser. No. 61/210,888, filed Mar. 23, 2009, and isa Continuation-in-Part (CIP) application of U.S. patent application Ser.No. 12/359,150, filed Jan. 23, 2009 (since issued as U.S. Pat. No.9,084,936), and is a Continuation of U.S. patent application Ser. No.11/999,475, filed Dec. 5, 2007 (since issued as U.S. Pat. No.9,108,107), and is a Continuation-in-Part (CIP) of application Ser. No.10/315,460, filed Dec. 10, 2002 (since issued as U.S. Pat. No.7,849,491), entitled, “APPARATUS AND METHOD FOR WIRELESS VIDEO GAMING,”each of which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates generally to the field of data processingsystems and particularly to an improved system, method, and graphicaluser interface for entering text with a physical or virtual gamecontroller.

BACKGROUND

The assignee of the present application has developed an onlineapplication streaming service described in the co-pending applicationsreferenced above. In operation, the application streaming serviceexecutes applications such as video games in response to control signalstransmitted over the Internet from client devices. The resulting videogenerated by an application is compressed and streamed to the clientdevices, then decompressed and rendered on the client devices at lowlatency, such that the user of the application has the perception thatthe controlled application is responding instantly.

The popularity of tablet-based client devices such as the Apple iPad™Samsung Galaxy,™ and HTC Puccini™ continues to grow. However, becauseuser input is restricted to the graphical display of these devices,tablets pose unique challenges, particularly when used for twitchlatency video games and other fast-action applications. The embodimentsof the invention described below address these challenges. While theseembodiments will be described within the context of an online streamingservice (such as described in the co-pending applications), it will beappreciated that the underlying principles of the invention are notlimited to use with an online streaming configuration.

In addition, entering text in a video game with a game controller iscurrently a cumbersome task. In a typical scenario, a user must navigatethrough a flat matrix of alphanumeric characters and select each letterindividually. Embodiments of the invention described below provideimproved techniques for entering text using a virtual or physical gamecontroller.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will be understood more fully from the detaileddescription that follows and from the accompanying drawings, whichhowever, should not be taken to limit the disclosed subject matter tothe specific embodiments shown, but are for explanation andunderstanding only.

FIGS. 1A-B illustrate a system architecture for executing online videogames according to one embodiment of the invention.

FIG. 2 illustrates different communication channels over which an onlinevideo game may be played in accordance with one embodiment of theinvention.

FIG. 3 illustrates one embodiment of a system architecture forcompressing audio/video generated by a video game.

FIGS. 4A-B illustrate a graphical element for a virtual thumb stickaccording to one embodiment of the invention.

FIGS. 5A-B illustrate a variety of different graphical elements for useon a tablet in accordance with one embodiment of the invention.

FIGS. 6-9 illustrate the graphical elements from FIGS. 4A-B and FIGS.5A-B in several different configurations for different games rendered ona tablet.

FIGS. 10A-10B illustrate physical elements of a game controller thatcorrespond to graphical user interface elements for a tablet provided ina configuration of a virtual controller.

FIGS. 11-16 illustrate a graphical user interface for entering textusing a virtual or physical controller.

DESCRIPTION OF EXAMPLE EMBODIMENTS

In the following description specific details are set forth, such asdevice types, system configurations, communication methods, etc., inorder to provide a thorough understanding of the present disclosure.However, persons having ordinary skill in the relevant arts willappreciate that these specific details may not be needed to practice theembodiments described.

The assignee of the present application has developed an online videogaming and application hosting system. Certain embodiments of thissystem are described, for example, in U.S. patent application Ser. No.12/538,077, filed Aug. 7, 2009, entitled SYSTEM AND METHOD FORACCELERATED MACHINE SWITCHING (hereinafter '077 application) whichclaims priority to U.S. Provisional Application Ser. No. 61/210,888,filed, Mar. 23, 2009, and is a continuation-in-part (CIP) application ofSer. No. 10/315,460 filed Dec. 10, 2002 entitled, “APPARATUS AND METHODFOR WIRELESS VIDEO GAMING”, which is assigned to the assignee of thepresent CIP application. These applications are sometimes referred to asthe “co-pending applications” and are incorporated herein by reference.A brief description of certain pertinent aspects of the online videogame and application hosting system described in the co-pendingapplications will now be provided, following by a detailed descriptionof a virtualization and encryption system and method for hostingapplications.

An Exemplary Online Video Game and Application Hosting System

FIG. 1A illustrates one embodiment of a video game/application hostingservice 110 described in the co-pending applications. The HostingService 210 hosts applications running on Servers 102, that accept inputfrom an Input device 121, received by Home or Office Client 115, andsent through the Internet 110 to Hosting Service 210. The Servers 102are responsive to the input, and update their video and audio outputaccordingly which is compressed through Low-Latency Video Compression104. The compressed video is then streamed through the Internet 110 tobe decompressed by the Home or Office Client 115, and then displayed onMonitor or SD/HDTV 122. This system is an low-latency streaminginteractive video system as more thoroughly described in theaforementioned “co-pending applications.”

FIG. 1B illustrates one embodiment of the invention in which the clientis a tablet-based client 175 such as an Apple iPad™, Samsung Galaxy,™ orHTC Puccini™. Control signals are 106 are generated in response to theuser manipulating graphical control elements on the display of thetablet (several embodiments of which are described below). The hostingservice 210 executes an application such as a video game in response tothe control signals 106 and generates video of the application. Thelow-latency video compression module 104 then compresses the video andstreams the compressed video to the tablet-based client 175. The lowlatency video decompression module 112 on the client decompresses andrenders the video on the tablet display 176. It should be noted that,while the term “tablet” is used herein for the sake of brevity, “tablet”is intended to be inclusive of devices of all sizes and categories thatare capable of implementing the functionality described herein, such asApple iOS devices inclusive of iPhone and iPod, Android devicesinclusive of smartphones, and computers or displays, all of thepreceding with or without buttons or controls, whether controls(inclusive of touch and motion-sensing controls) are built-in,physically attached, or wirelessly attached to the device.

As shown in FIG. 2, the network connection between the Hosting Service210 Home and Office Client 215 or tablet-based client 175 may beimplemented through a wide range of network technologies, of varyingdegrees of reliability, such as wired or optical fiber technologies thatare typically more reliable and wireless technologies that may besubject to unpredictable interference or range limitations (e.g. Wi-Fi)and are typically less reliable. Any of these client devices may havetheir own user input devices (e.g., keyboards, buttons, touch screens,track pads or inertial-sensing wands, video capture cameras and/ormotion-tracking cameras, etc.), or they may use external input devices221 (e.g., keyboards, mice, game controllers, inertial sensing wand,video capture cameras and/or motion tracking cameras, etc.), connectedwith wires or wirelessly. As described in greater detail below, thehosting service 210 includes servers of various levels of performance,including those with high-powered CPU/GPU processing capabilities.During playing of a game or use of an application on the hosting service210, a home or office client device 215 receives keyboard and/orcontroller input from the user. A tablet-based client 175 receives inputas the user manipulates graphical control elements on the tablet displayor moves the tablet so as to effect built-in motion sensors (in additionto what is shown in FIG. 2, tablets may also receive input from (but notlimited to) other built-in controls, such as keyboard, mouse, buttons,wheels, trackpads, joysticks, audio or video input, motion capture,etc., as well as external controls, such a wired or wireless-connectedgamepad, keyboard, mouse, trackpad, audio or video input,motion/position capture/sensing, etc.). In either case, the clients 215,175 transmit the control input (also called herein the “controllerinput”) through the Internet 206 to the hosting service 210 thatexecutes the gaming program code in response and generates successiveframes of video output (a sequence of video images) for the game orapplication software (e.g., if the user presses a button which woulddirect a character on the screen to move to the right, the game programwould then create a sequence of video images showing the charactermoving to the right). This sequence of video images is then compressedusing a low-latency video compressor, and the hosting service 210 thentransmits the low-latency video stream through the Internet 206. Thehome or office client device then decodes the compressed video streamand renders the decompressed video images on a monitor or TV.Consequently, the computing and graphical hardware requirements of theclient devices 215, 175 are significantly reduced. The client 215 onlyneeds to have the processing power to forward the keyboard/controllerinput to the Internet 206 and decode and decompress a compressed videostream received from the Internet 206, which virtually any personalcomputer is capable of doing today in software on its CPU (e.g., a IntelCorporation Core Duo CPU running at approximately 2 GHz is capable ofdecompressing 720p HDTV encoded using compressors such as H.264 andWindows Media VC9). And, in the case of any client devices 215, 175,dedicated chips can also perform video decompression for such standardsin real-time at far lower cost and with far less power consumption thana general-purpose CPU such as would be required for a modern PC.Notably, to perform the function of forwarding controller input anddecompressing video, client devices 215, 175 typically do not requireany specialized graphics processing units (GPUs), optical drive or harddrives.

As games and applications software become more complex and morephoto-realistic, they will require higher-performance CPUs, GPUs, moreRAM, and larger and faster disk drives, and the computing power at thehosting service 210 may be continually upgraded, but the end user willnot be required to update the home or office client platform 215 ortablet-based client 175 since its processing requirements will remainconstant for a display resolution and frame rate with a given videodecompression algorithm. Thus, the hardware limitations andcompatibility issues seen today do not exist in the system illustratedin FIG. 2.

Further, because the game and application software executes only inservers in the hosting service 210, there never is a copy of the game orapplication software (either in the form of optical media, or asdownloaded software) in the user's home or office (“office” as usedherein unless otherwise qualified shall include any non-residentialsetting, including, schoolrooms, for example). This significantlymitigates the likelihood of a game or application software beingillegally copied (pirated), as well as mitigating the likelihood of avaluable database that might be used by a game or applications softwarebeing pirated. Indeed, if specialized servers are required (e.g.,requiring very expensive, large or noisy equipment) to play the game orapplication software that are not practical for home or office use, theneven if a pirated copy of the game or application software wereobtained, it would not be operable in the home or office.

FIG. 3 illustrates an embodiment of components of a server center forhosting service 210 utilized in the following feature descriptions.Inbound internet traffic 301 from user clients such as the tablet-basedclient 175 mentioned above is directed to inbound routing 301.Typically, inbound internet traffic 301 will enter the server center viaa high-speed fiber optic connection to the Internet, but any networkconnection means of adequate bandwidth, reliability and low latency willsuffice. Inbound routing 302 is a system of network (the network can beimplemented as an Ethernet network, a fiber channel network, or throughany other transport means) switches and routing servers supporting theswitches which takes the arriving packets and routes each packet to theappropriate application/game (“app/game”) server 321-325.

In one embodiment, a packet which is delivered to a particular app/gameserver represents a subset of the data received from the client and/ormay be translated/changed by other components (e.g., networkingcomponents such as gateways and routers) within the data center. In somecases, packets will be routed to more than one server 321-325 at a time,for example, if a game or application is running on multiple servers atonce in parallel. RAID arrays 311-312 are connected to the inboundrouting network 302, such that the app/game servers 321-325 can read andwrite to the RAID arrays 311-312. Further, a RAID array 315 (which maybe implemented as multiple RAID arrays) is also connected to the inboundrouting 302 and data from RAID array 315 can be read from app/gameservers 321-325. The inbound routing 302 may be implemented in a widerange of prior art network architectures, including a tree structure ofswitches, with the inbound internet traffic 301 at its root; in a meshstructure interconnecting all of the various devices; or as aninterconnected series of subnets, with concentrated traffic amongstintercommunicating device segregated from concentrated traffic amongstother devices. One type of network configuration is a SAN which,although typically used for storage devices, it can also be used forgeneral high-speed data transfer among devices. Also, the app/gameservers 321-325 may each have multiple network connections to theinbound routing 302. For example, a server 321-325 may have a networkconnection to a subnet attached to RAID Arrays 311-312 and anothernetwork connection to a subnet attached to other devices. While theshared hardware compression 330 is an exemplary embodiment to compressthe video output of the app/game Servers 321-325 may have completelyindependent or partially independent video compression capability thatmay be implemented in dedicated hardware or in software or in somecombination thereof, and the video may be compressed by one or aplurality of compression means using one or more compression algorithms,in either unidirectional or bi-directional form, adaptively ornon-adaptively, etc. The descriptions herein that reference sharedhardware compression 330 also may be applied, as appropriate, to theabove referenced compression embodiments.

The app/game servers 321-325 may all be configured the same, somedifferently, or all differently, as previously described in relation toservers 102 in the embodiment illustrated in FIGS. 1A-B. In oneembodiment, each user, when using the hosting service is typically usingat least one app/game server 321-325. For the sake of simplicity ofexplanation, we shall assume a given user is using app/game server 321,but multiple servers could be used by one user, and multiple users couldshare a single app/game server 321-325. The user's control input, sentfrom a client such as a tablet-based client 175 mentioned above isreceived as inbound Internet traffic 301, and is routed through inboundrouting 302 to app/game server 321. App/game server 321 uses the user'scontrol input as control input to the game or application running on theserver, and computes the next frame of video and the audio associatedwith it. App/game server 321 then outputs the uncompressed video/audio329 to shared video compression 330. App/game server may output theuncompressed video via any means, including one or more Gigabit Ethernetconnections, but in one embodiment the video is output via a DVIconnection and the audio and other compression and communication channelstate information is output via a Universal Serial Bus (USB) connection.

The shared video compression 330 compresses the uncompressed video andaudio from the app/game servers 321-325. The compression maybeimplemented entirely in hardware, or in hardware running software. Theremay a dedicated compressor for each app/game server 321-325, or if thecompressors are fast enough, a given compressor can be used to compressthe video/audio from more than one app/game server 321-325. For example,at 60 fps a video frame time is 16.67 ms. If a compressor is able tocompress a frame in 1 ms, then that compressor could be used to compressthe video/audio from as many as 16 app/game servers 321-325 by takinginput from one server after another, with the compressor saving thestate of each video/audio compression process and switching context asit cycles amongst the video/audio streams from the servers. This resultsin substantial cost savings in compression hardware. Since differentservers will be completing frames at different times, in one embodiment,the compressor resources are in a shared pool 330 with shared storagemeans (e.g., RAM, Flash) for storing the state of each compressionprocess, and when a server 321-325 frame is complete and ready to becompressed, a control means determines which compression resource isavailable at that time, provides the compression resource with the stateof the server's compression process and the frame of uncompressedvideo/audio to compress.

Note that part of the state for each server's compression process mayinclude information about the compression itself, such as the previousframe's decompressed frame buffer data which may be used as a referencefor P tiles, the resolution of the video output; the quality of thecompression; the tiling structure; the allocation of bits per tiles; thecompression quality, the audio format (e.g., stereo, surround sound,Dolby® AC-3). But the compression process state may also includecommunication channel state information regarding the peak data rate andwhether a previous frame is currently being output (and as result thecurrent frame should be ignored), and potentially whether there arechannel characteristics which should be considered in the compression,such as excessive packet loss, which affect decisions for thecompression (e.g., in terms of the frequency of I tiles, etc). As thepeak data rate or other channel characteristics change over time, asdetermined by an app/game server 321-325 supporting each user monitoringdata sent from the tablet-based client 175 (or other type of client),the app/game server 321-325 may send the relevant information to theshared hardware compression 330.

The shared hardware compression 330 may also packetize the compressedvideo/audio using means such as those previously described, and ifappropriate, applying FEC codes, duplicating certain data, or takingother steps to as to adequately ensure the ability of the video/audiodata stream to be received by the tablet-based client 175 (or other typeof client) and decompressed with as high a quality and reliability asfeasible.

Some applications, such as those described below, require thevideo/audio output of a given app/game server 321-325 to be available atmultiple resolutions (or in other multiple formats) simultaneously. Ifthe app/game server 321-325 so notifies the shared hardware compression330 resource, then the uncompressed video/audio 329 of that app/gameserver 321-325 will be simultaneously compressed in different formats,different resolutions, and/or in different packet/error correctionstructures. In some cases, some compression resources can be sharedamongst multiple compression processes compressing the same video/audio(e.g., in many compression algorithms, there is a step whereby the imageis scaled to multiple sizes before applying compression. If differentsize images are required to be output, then this step can be used toserve several compression processes at once). In other cases, separatecompression resources will be required for each format. In any case, thecompressed video/audio 339 of all of the various resolutions and formatsrequired for a given app/game server 321-325 (be it one or many) will beoutput at once to outbound routing 340. In one embodiment the output ofthe compressed video/audio 339 is in UDP format, so it is aunidirectional stream of packets.

The outbound routing network 340 comprises a series of routing serversand switches which direct each compressed video/audio stream to theintended user(s) or other destinations through outbound Internet traffic399 interface (which typically would connect to a fiber interface to theInternet) and/or back to the delay buffer 315 (implemented as a RAIDarray in one embodiment), and/or back to the inbound routing 302, and/orout through a private network (not shown) for video distribution. Notethat (as described below) the outbound routing 340 may output a givenvideo/audio stream to multiple destinations at once. In one embodiment,this is implemented using Internet Protocol (IP) multicast in which agiven UDP stream intended to be streamed to multiple destinations atonce is broadcasted, and the broadcast is repeated by the routingservers and switches in the outbound routing 340. The multipledestinations of the broadcast may be to multiple users' clients 175 viathe Internet, to multiple app/game servers 321-325 via inbound routing302, and/or to one or more delay buffers 315. Thus, the output of agiven server 321-322 is compressed into one or multiple formats, andeach compressed stream is directed to one or multiple destinations.

Further, in another embodiment, if multiple app/game servers 321-325 areused simultaneously by one user (e.g., in a parallel processingconfiguration to create the 3D output of a complex scene) and eachserver is producing part of the resulting image, the video output ofmultiple servers 321-325 can be combined by the shared hardwarecompression 330 into a combined frame, and from that point forward it ishandled as described above as if it came from a single app/game server321-325.

Note that in one embodiment, a copy (in at least the resolution orhigher of video viewed by the user) of all video generated by app/gameservers 321-325 is recorded in delay buffer 315 for at least some periodof time (15 minutes in one embodiment). This allows each user to“rewind” the video from each session in order to review previous work orexploits (in the case of a game). Thus, in one embodiment, eachcompressed video/audio output 339 stream being routed to a client 175 isalso being multicasted (in another embodiment, a separate unicasttransmissions is used) to a delay buffer 315. When the video/audio isstored on a delay buffer 315, a directory on the delay buffer 315provides a cross reference between the network address of the app/gameserver 321-325 that is the source of the delayed video/audio and thelocation on the delay buffer 315 where the delayed video/audio can befound.

App/game servers 321-325 may not only be used for running a givenapplication or video game for a user, but they may also be used forcreating the user interface applications for the hosting service 210that supports navigation through hosting service 210 and other features.

System, Method, and Graphical User Interface for Controlling anApplication on a Tablet

As mentioned above, in one embodiment of the invention, a tablet-basedclient 175 is used which includes a set of graphical user interfaceelements to allow the user to navigate and control the operation of anonline application such as a video game. In one embodiment, the set ofgraphical user input elements include elements which correspond tocommon buttons, knobs and joysticks on a game controller, one example ofwhich is illustrated in FIGS. 10A-B. Specifically, the physicalcontroller includes left and right joysticks 101 and 102, respectively,which may be freely moved in all directions to perform game inputfunctions. For example, in an adventure game or first person shooter,the joysticks may be manipulated to move a character throughout thegame. Also shown in FIGS. 10A-B is a directional pad (commonly referredto as a D-pad) 108 to perform up, down, left, and right motion and/orcursor controls. A group of four action buttons 105 are provided toperform various application-specific functions (e.g., jump, run, switchweapons, etc). A set of transport buttons 104 positioned towards thebottom of the controller allow the user to perform play/pause, stop,fast forward, and rewind operations from within a game (e.g., to pause,fast forward, rewind, etc, within the game). A set of three navigationbuttons 103 provided towards the middle of the controller may providevarious different navigation functions and/or other system-specificfunctions. For example, in one embodiment, the central circular buttoncauses a user navigation interface to appear overlaid on top of thecurrent game; the left button is used to back out of menu items; and theright button is used to enter/select menu items. Of course, theunderlying principles of the invention are not limited to any particularset of button functions. Bumper buttons 106 a-b (sometimes referred toas shoulder buttons) and trigger buttons 107 a-b provide variouswell-known game specific functions (e.g., fire, change weapon, reload,etc).

An exemplary set of graphical user interface elements which providetouch-screen functionality on a tablet-based computer is illustrated inFIGS. 4A-B and FIGS. 5A-B, and exemplary arrangements for thesegraphical user interface elements in a series of video games isillustrated in FIGS. 6-9. Turning first to FIGS. 4A-B, one embodiment ofa virtual controller 405 is shown which is manipulated by a user'sthumbs (or other fingers) on the surface of the tablet-based client(i.e., in a similar manner as the left and right joysticks 101 and 102,respectively, are manipulated on the physical controller). In oneembodiment, the inner circle 405 is the controller which, in oneembodiment, provides omnidirectional, free form movement (i.e., notlimited to circular or XY movement). The outer circle 406 identifies theborder perimeter, i.e., the maximum distance that the controller 405 maybe moved from its starting position (as shown in FIG. 4B).

In one embodiment, a user's finger is not limited to the space definedby the border perimeter 406. Rather, the user may move a finger outsideof this region. In one embodiment, when a user's finger is moved outsideof the border perimeter, the virtual controller 405 travels on theborder 406 (as shown in FIG. 4B). While the user may move a fingeroutside of the region defined by the border perimeter, the user may onlymove a finger within a defined region for each virtual (sometimesreferred to herein as a “hit box”) controller, such as an upper right orlower left quadrant as indicated by arrows 610, 710-711, 810, and910-911 in FIGS. 6-9.

In one embodiment, different modes of operation may be specified for thevirtual controller 405 including a “relative mode” (also referred toherein as “sloppy thumbsticks”) and an “absolute mode.” In oneembodiment, when in relative mode, the virtual controller 405 isinitially drawn at the point the user places a thumb (or other finger)on the tablet display. The movement of an in-game object using thevirtual controller then relates to the movement of finger relative tothis initial touch point. In one embodiment, if the user lifts and thenplaces his/her thumb back on a new position on the display, the firstvirtual controller 601 will be redrawn at the new position. In contrast,when in absolute mode, the virtual controller is drawn in a fixedlocation on the display and is moved based on a user's finger positionon screen (as described above). One embodiment of the invention allowsthe user to select between the relative and absolute modes describedabove.

In one embodiment, the size of the virtual controller and the definedregion in which the virtual controller can be manipulated by the user(the “hit box”) is set automatically, based on the dimensions of thetablet computer's display. The number and diameter of each virtualcontroller may be configured per game. For example, in FIG. 6, arelatively larger virtual controller 601 is configured in the lower leftquadrant of the display while a relatively smaller virtual controller602 is configured in the upper right quadrant of the display.

Additionally, in one embodiment, the mode of each virtual controller maybe configured on a per-game or per-user basis. For example, in FIG. 6,the first virtual controller 601 is configured as a “sloppy” or“relative mode” controller, whereas the second controller 602 isconfigured as a “non-sloppy” or “absolute mode” controller. Thus, duringgameplay, the first virtual controller will be automatically drawnwherever the user places a thumb or other finger within the lower leftquadrant. The movement of the virtual controller is then relative tothis initial touch point.

By contrast, the second virtual controller 602, which is operated as a“non-sloppy” or “absolute mode” controller is drawn in the same locationon the display regardless of the user's initial finger positioning.Different combinations of relative/absolute mode and large/smallcontrollers may be specified for each game (e.g., by the game designer)and/or may be configurable by the end user or the hosting service 210.

In one embodiment, the virtual controllers 405 and other graphical userinterface elements are transparent or semi-transparent so that the usercan see the results of the game being played beneath. The virtualcontrollers and other user interface elements may also be in a low dimstate when not used and then may be brightened in response to a user'stouch. Alternatively, or in addition, the transparency level of thecontroller and other user interface elements may change in response to auser's touch. In one embodiment, each of the individual user interfaceelements including the controller may have unique dimness ortransparency settings (e.g., as specified by the user), but default toglobal settings.

In one embodiment, the sensitivity of each virtual controller is alsoconfigurable by the game designer, the end user, and/or the hostingservice 210. For example, a first sensitivity setting may cause thevirtual controller 405 to move with a different amount of accelerationin response to the movement of a user's finger on the display of thetablet.

Additional graphical user interface elements for a tablet whichcorrespond to physical elements on a game controller (such as shown inFIGS. 10A-B) are shown in FIGS. 5A-B:

Select 501 mimics a select button on a physical game controller. Thisgraphical button may be optional and used in different configurable waysfor each game or other application.

Dashboard 502 launches a graphical navigation menu (referred to as the“dashboard”) provided by the online hosting service 210. In oneembodiment, the graphical navigation menu is illustrated in FIGS. 16-23of the co-pending U.S. patent application Ser. No. 12/538,077, filedAug. 7, 2009, entitled SYSTEM AND METHOD FOR ACCELERATED MACHINESWITCHING, which has been incorporated herein by reference.

Start 503 provides games options menu/pause function and is used for allgames streamed by the hosting service 210.

Record 504 allows a user to record gameplay for a game currently beingplayed (referred to as “brag clips” in the co-pending applications). Inone embodiment, the recordings may be stored on the hosting service 210and played back by the user and other users.

A “touch and tap” user interface element 505 allows finger dragging andtaping on the screen of the tablet. In one embodiment, the touch and tapelement 505 is provided on mouse-click style games (e.g., World of Goo™designed by 2D Boy™).

A variety of action buttons are provided which correspond to the actionbuttons on a typical game controller (such as that illustrated in FIGS.10A-B). These include a right bumper 511, left bumper 512, right trigger513, left trigger 514, and X 523, Y 521, A 522, and B 524 buttons(corresponding to the X, Y, A, and B buttons 105 on the physicalcontroller shown in FIG. 10A). Various combinations of the actionbuttons and their placement may be customized for each game, by the gamedesigner, the end user, and/or the hosting service 210. While some ofthese customizations are shown below with respect to FIGS. 6-9, theseare merely illustrative examples which should not be read to limit thescope of the present invention.

A virtual keyboard button 515 in FIG. 5B opens a virtual keyboard on thetablet. In one embodiment, the keyboard opened is the native virtualkeyboard designed for the particular tablet being used and opensautomatically when text input is required.

Left slider trigger 516 and right slider trigger 525 are graphical userinterface buttons which can be configured to reset to zero when theuser's finger stops touching (similar to a Gamepad trigger) or can beconfigured to stay at a current value set when the user's finger stopstouching. Additionally, these buttons may be angled vertically orhorizontally.

The labels button 526 may be used to toggle between showing/hidinglabels for each of the game controls. By way of example, in FIG. 8B, aplurality of labels are displayed for a tennis game (i.e., Y=Lob,B=Slice, A=Top Spin, and the virtual controller=move). In oneembodiment, certain labels are not affected by the labels button (e.g.,the dashboard, start, select, etc.)

Turning now to some of the specific configurations set forth in FIGS.6-9, FIG. 6 illustrates one particular game for which a first virtualcontroller 601 is positioned in the lower left quadrant of the tablet'sdisplay 600 and a second virtual controller 602 is positioned in theupper right quadrant of the tablet's display 600. The first virtualcontroller 601 in the illustrated example is configured for a relativemode of operation (i.e., sloppy), whereas the second virtual controller602 is configured for an absolute mode of operation (“non-sloppy”).Thus, as previously described, the first controller 601 is drawn at aposition where the user places a finger whereas the second controller602 is drawn in a fixed position on the display. A region is defined inthe upper right portion of the display (as indicated by arrows 610) intowhich the user must place a finger in order to operate the secondvirtual controller 602. A similar region may also be defined withrespect to the first controller 601 or, alternatively, the secondcontroller may be operable anywhere on the left half of the display(i.e., with the second controller being drawn wherever the user places afinger from his/her left hand while holding the tablet-based client).

Towards the right side of the display is a virtual right trigger (RT)button 603 and a set of X, Y, A, B buttons 604 which have variousdefined functions depending on the video game being played. Towards thebottom of the display are a virtual keyboard button 605 (for enteringtext), a dashboard button 606 (for opening the graphical user interfaceof the hosting service), a start button 607 (for providing game optionsand menu/pause functions), and a record button 608 (for recording bragclips as described above).

FIG. 7 illustrates another embodiment in which a first virtualcontroller 701 is positioned in the lower left quadrant of the tablet'sdisplay 600 and a second virtual controller 702 is positioned in theupper right quadrant of the tablet's display. As in the embodiment shownin FIG. 6, the first virtual controller 701 is configured for a relativemode of operation (i.e., sloppy), whereas the second virtual controller702 is configured for an absolute mode of operation (“non-sloppy”). Inthis embodiment, regions are defined in both the lower left and upperright portions of the display (as indicated by arrows 701 and 702,respectively) into which the user must place a finger in order tooperate the first virtual controller 701 and the second virtualcontroller 702, respectively.

Towards the right side of the display is a virtual right bumper (RB)button 703 and a set of X, Y, A, B buttons 704 which have variousdefined functions depending on the video game being played. Towards thebottom of the display are a virtual keyboard button 705 (for enteringtext), a dashboard button 706 (for opening the graphical user interfaceof the hosting service), a start button 707 (for providing game optionsand menu/pause functions), and a record button 708 (for recording bragclips as described above).

FIGS. 8A-B illustrate another embodiment in which a first virtualcontroller 801 configured for a relative mode of operation (i.e.,sloppy) is displayed in the lower left corner of the tablet's display.This embodiment does not use a second virtual controller. As previouslymentioned, the specific configuration of the virtual controllers andother input elements may be specified by the game designer, end user,and/or hosting service 210. Once again, a region is defined in the lowerleft corner of the display within the user must place a finger toactivate the virtual controller 801 (as defined by lines 810).

Other elements shown in FIGS. 8A-B include a left bumper 803 and set ofY, A, and B control elements 804 displayed on the right side of thetablet's display (so as to be accessible by the user's right hand whenholding the tablet). Towards the bottom of the display are a virtualkeyboard button 805 (for entering text), select button 806 (formimicking the select button on a physical game controller), a dashboardbutton 807 (for opening the graphical user interface of the hostingservice), a start button 808 (for providing game options and menu/pausefunctions), and a record button 809 (for recording brag clips asdescribed above).

In FIG. 8B, labels are displayed next to the virtual controller 801 andthe action buttons 803, 804, to instruct the user how to operate thevirtual controls. As previously mentioned, the labels may be toggled onand off using a separate “labels” button. Alternatively, or in addition,the labels may appear automatically at specified times (e.g., at thebeginning of a game, at times when no action is occurring in the gamesuch as when one user scores a point, etc).

FIG. 9 illustrates yet another embodiment in which a first virtualcontroller 901 set to an absolute mode of operation (“non-sloppy”) ispositioned in the lower left quadrant of the tablet's display (in aregion defined by lines 910) and a second virtual controller 902 alsoset to an absolute mode of operation is positioned in the upper rightquadrant of the tablet's display (in a region defined by lines 911).Towards the right side of the display is a virtual left trigger 903 anda right slider trigger 925. As mentioned, the slider trigger may beconfigured to reset to zero when the user's finger stops touching(similar to a Gamepad trigger) or may be configured to stay at a currentvalue set when the user's finger stops touching. Also illustrated to theright on the tablet's display are a set of X, Y, A, and B buttons 904which have various defined functions depending on the video game beingplayed.

Towards the bottom of the display are the virtual keyboard button 905(for entering text), a dashboard button 906 (for opening the graphicaluser interface of the hosting service), a start button 907 (forproviding game options and menu/pause functions), and a record button908 (for recording brag clips as described above).

In one embodiment, the configuration of the virtual controller and othergraphical user interface elements is stored in a database on the hostingservice 210 (e.g., accessible via RAID array 105). Each video game orapplication hosted by the hosting service 210 is provided with its owndefault graphical user interface configuration. In one embodiment, eachindividual user of the hosting service 210 may make user-specificadjustments to the default configuration and store the adjustmentsstored in the database. For example, the user may modify features suchas the location of the virtual controller and the virtual controllermode of operation (e.g., sloppy vs. non-sloppy).

When a client connects to a specific application or video game on thehosting service 210 the virtual user interface configuration is sentfrom the database, through the hosting service 210 down to thetablet-based client. In one embodiment, the virtual controllerconfiguration is stored as a JavaScript Object Notation (JSON) string.However, the underlying principles of the invention are not limited toany particular type of program code.

When a tablet-based client 175 receives the graphical user interfaceconfiguration from the hosting service 210, it will render theuser-specific one (if one exists) or the default one. In one embodiment,a user may upload new user-specific virtual pad configurations to thehosting service 210 and it will be stored in the database for that userfor that game.

In one embodiment, the graphical user interface elements are stored inthe hosting service database as a set of widgets (a small executablewith graphical and interactive elements) with configuration data foreach application or video game defining where the widget should bedrawn, which widget to draw, how big to draw it (scale), the size of thehitbox (independent from the actual icon/image), text (if a label isused), specific configuration parameters for that widget (such assensitivity for a thumbstick widget), and the action to take when theuser interacts with widget. For example, each widget may be associatedwith a particular physical controller action (e.g., send a left mouseclick, press the j-button on a keyboard, or press the right trigger on agamepad, etc). In one embodiment the widgets execute locally on thetablet. In another embodiment the widgets execute remotely in hostingservice 210. In yet another embodiment, parts of a widget execute ineach the tablet and the hosting service 210.

Thus, in the examples described above, similar types of controls aredisplayed in consistent positions across different games, making iteasier for the user to learn how to operate various games provided bythe hosting service 210 on a tablet-based client.

Embodiments of the Invention for Entering Text Using a Virtual orPhysical Game Controller

One embodiment of the invention comprises a graphical user inputallowing text to be entered efficiently in a video game or otherapplication using nothing but a virtual or physical game controller. Inthis embodiment, the left thumbstick and A, B, X, and Y action buttonsare used to quickly access the letters of the keyboard.

FIG. 11 illustrates the graphical user interface employed in thisembodiment which includes a 3×3 array of selectable boxes 1101 forselecting text (although the underlying principles of the invention arenot limited to any particular number of boxes). In operation, ahighlight element 1103 is moved from one box to the next via the leftthumbstick of the game controller 101. The center box is shownhighlighted in FIG. 11. When a particular box having a desiredalphanumeric character is highlighted, A, B, X, and Y elements 1105appear on each edge of the box as shown. A user may then quickly selecta particular alphanumeric character within the highlighted box byselecting the A, B, X, or Y button on the controller corresponding tothat alphanumeric character.

In the particular example shown in FIG. 11, selecting the A key willcause the displayed underscore character to be entered in the text entryregion 1106, selecting the X key will generate a backspace within thetext entry region 1106, selecting the Y key will update the set of boxeswith all capital characters (see FIG. 13), and selecting the B key willupdate the set of boxes with numbers and non-alphabetic characters(e.g., numbers and symbols as shown in FIG. 14).

In one embodiment, the positioning of the A, B, X, and Y elements 1105as each box is highlighted. Thus, using the left thumbstick, if the usermoves the highlight element to box 1102 in FIG. 11, the X element wouldselect the letter “a,” the Y element would select the letter “b,” the Belement would select the letter “c,” and the A element would select theletter “d.”

In one embodiment, a “cancel” button 1107 may be selected via thethumbstick to cancel the text entry GUI (e.g., by moving the highlightelement to the right from the right side of the array of boxes), and a“done” button 1108 may similarly be selected when text entry iscomplete. Note that the text entry region 1106 in the example shown inFIG. 11 is for entering a password.

As indicated towards the bottom of FIG. 11, other buttons on the gamecontroller may be assigned for further simplify text entry. Here, theleft bumper is assigned a move left function, the right bumper isassigned a move right function, the left trigger is assigned a shiftfunction (e.g., to shift between lower case, upper case, andnumbers/symbols within the array of boxes), and the right bumper isassigned a return function.

FIG. 12 illustrates another example in which a comment is being enteredin the text entry region of the display. Other buttons shown in FIG. 12may be assigned the same functions as in FIG. 11.

FIG. 13 illustrates an example in which the user has previously selectedlowercase characters (e.g., by selecting Y from the center box shown inFIG. 11 or by selecting the left trigger (shift) key). Once selected,the center box was repopulated with a similar set of selectable options,the only difference being that a upper case option is now selectable viathe Y key (as indicated by the upper case graphic “ABC”). Maintainingthe center box in this consistent manner makes it easier for the enduser to remember how to navigate between the different alphanumericcharacter sets.

FIG. 14 illustrates an embodiment in which numbers and symbols have beenselected resulting in the array of boxes being populated with numbersand symbols (e.g., by selecting the B key from the GUI shown in FIG.13). Once again, the center box is repopulated in a consistent manner toease navigation. The remaining portions of the GUI have not changed.

FIG. 15 illustrates an embodiment in which the typing has resulted in alist being generated within the text entry region 1106. In this case,any entries starting with the first set of entered characters (“alit”)are placed at the top of the list, followed by entries containing thecharacter string (“alit”). The user may then move the highlight elementdown to select entries in the list and select an element (e.g., usingthe right trigger button or other selection button).

In FIG. 16, the user has selected the first entry in the text entryregion 1106. The user may thereafter move the highlighted down to thedone key to select the highlighted entry.

In one embodiment, the various functional modules illustrated herein andthe associated steps may be performed by specific hardware componentsthat contain hardwired logic for performing the steps, such as anapplication-specific integrated circuit (“ASIC”) or by any combinationof programmed computer components and custom hardware components.

In one embodiment, the modules may be implemented on a programmabledigital signal processor (“DSP”) such as a Texas Instruments' TMS320xarchitecture (e.g., a TMS320C6000, TMS320C5000, . . . etc). Variousdifferent DSPs may be used while still complying with these underlyingprinciples.

Embodiments may include various steps as set forth above. The steps maybe embodied in machine-executable instructions which cause ageneral-purpose or special-purpose processor to perform certain steps.Various elements which are not relevant to these underlying principlessuch as computer memory, hard drive, input devices, have been left outof some or all of the figures to avoid obscuring the pertinent aspects.

Elements of the disclosed subject matter may also be provided as amachine-readable medium for storing the machine-executable instructions.The machine-readable medium may include, but is not limited to, flashmemory, optical disks, CD-ROMs, DVD ROMs, RAMs, EPROMs, EEPROMs,magnetic or optical cards, propagation media or other type ofmachine-readable media suitable for storing electronic instructions. Forexample, the present invention may be downloaded as a computer programwhich may be transferred from a remote computer (e.g., a server) to arequesting computer (e.g., a client) by way of data signals embodied ina carrier wave or other propagation medium via a communication link(e.g., a modem or network connection).

It should also be understood that elements of the disclosed subjectmatter may also be provided as a computer program product which mayinclude a machine-readable medium having stored thereon instructionswhich may be used to program a computer (e.g., a processor or otherelectronic device) to perform a sequence of operations. Alternatively,the operations may be performed by a combination of hardware andsoftware. The machine-readable medium may include, but is not limitedto, floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks,ROMs, RAMs, EPROMs, EEPROMs, magnet or optical cards, propagation mediaor other type of media/machine-readable medium suitable for storingelectronic instructions. For example, elements of the disclosed subjectmatter may be downloaded as a computer program product, wherein theprogram may be transferred from a remote computer or electronic deviceto a requesting process by way of data signals embodied in a carrierwave or other propagation medium via a communication link (e.g., a modemor network connection).

Additionally, although the disclosed subject matter has been describedin conjunction with specific embodiments, numerous modifications andalterations are well within the scope of the present disclosure.Accordingly, the specification and drawings are to be regarded in anillustrative rather than a restrictive sense.

1. A method, comprising: receiving selection of an online application ata hosting service, the selection received from a remote client device;executing an instance of the online application on a server of thehosting service, in response to the selection, the execution generatingframes of streaming video content for the online application for onwardtransmission to the remote client device; identifying graphical userinterface elements used for interacting with the streaming video contentof the online application, wherein at least a part of the graphical userinterface elements are executing on the server of the hosting service;and encoding the graphical user interface elements for onwardtransmission to the remote client device for rendering, wherein theencoded graphical user interface elements are decoded at the remoteclient device and rendered on a display of the remote client device, thegraphical user interface elements map to at least some controls of aphysical game controller, and wherein inputs provided at each of thegraphical user interface elements rendered at the remote client deviceare interpreted by the server of the hosting service to control a stateof the online application.
 2. The method of claim 1, wherein thegraphical user interface elements are identified in part by retrievingvirtual user interface configuration defined for the online application,the virtual user interface configuration including the graphical userinterface elements used for interacting with the online application. 3.The method of claim 2, wherein the virtual user interface configurationis configured to include select ones of the graphical user interfaceelements based on a state of the online application.
 4. The method ofclaim 2, wherein the virtual user interface configuration is encodedwith the frames of the streaming video content, the virtual userinterface configuration includes a user interface defined for thedisplay of the remote client device, the user interface used forrendering the graphical user interface elements.
 5. The method of claim1, wherein the graphical user interface elements are decoded at theremote client device and rendered as an overlay over the frames of thestreaming video content.
 6. The method of claim 5, wherein the graphicaluser interface elements are configured to be transparent orsemi-transparent so as to allow viewing of the streaming video contentrendering beneath the overlay.
 7. The method of claim 1, wherein eachone of the graphical user interface elements is configured to includeone or more distinct settings, the one or more distinct settings beingspecified by a user viewing the streaming video content, or by adeveloper of the online application, or by the hosting service.
 8. Themethod of claim 7, wherein the one or more distinct settings specifiedfor each graphical user interface element includes a first setting foran inactive state and a second setting for an active state, and whereinthe one or more distinct settings include activation setting, ortransparency setting, or brightness setting, or sensitivity setting, orcolor setting, or scale setting, or size setting, or location setting,or operational mode setting, or operational direction setting, orgraphical user interface element type setting, or any two or morecombinations thereof.
 9. The method of claim 7, wherein the one or moredistinct settings are configured for the display of the remote clientdevice on which the streaming video content and the graphical userinterface elements are rendered.
 10. The method of claim 1, wherein thegraphical user interface elements are defined as a set of widgets,wherein each widget in the set of widgets is associated with aparticular graphical user interface element configured to provideparticular control action of the physical game controller.
 11. Themethod of claim 10, wherein a part of each widget executes locally atthe remote client and a remaining part executes on the server of thehosting service.
 12. The method of claim 1, wherein the onlineapplication is a video game, and a number and types of the graphicaluser interface elements are based on types of control actions needed forinteracting with streaming video content of the video game.
 13. Themethod of claim 1, wherein the graphical user interface elements arespecific for the online application and for the display of the remoteclient device.
 14. The method of claim 1, wherein the graphical userinterface elements include a virtual controller with controls that mimiccontrols of a physical controller.
 15. The method of claim 14, whereinthe virtual controller includes user specific adjustments provided by auser during previous interactive sessions of the online application. 16.The method of claim 15, wherein the user specific adjustment identifiesconfiguration parameters used for rendering the virtual controller onthe user interface rendered at the remote client device.
 17. The methodof claim 1, wherein the graphical user interface elements include userspecific adjustments provided by a user during previous interactivesessions of the online application, wherein the user specificadjustments identifies configuration parameters used for rendering thegraphical user interface elements on the display of the remote clientdevice of the user.
 18. The method of claim 1, further includes,receiving user specific adjustment to one or more of the graphical userinterface elements from a user during a current session, the adjustmentto the one or more of the graphical user interface elements updated to avirtual user interface configuration stored in a database of the hostingservice and used during subsequent sessions of the online application.19. A non-transitory computer readable storage medium having programinstructions, which when executed by a processor of a computing deviceof a hosting service performs a method, the program instructionscomprising: program instructions for receiving selection of an onlineapplication at a hosting service, the selection received from a remoteclient device; program instructions for executing an instance of theonline application on a server of the hosting service, in response tothe selection, the execution generating frames of streaming videocontent for the online application for onward transmission to the remoteclient device; program instructions for identifying graphical userinterface elements used for interacting with the streaming video contentof the online application, wherein at least a part of the graphical userinterface elements are executing on the server of the hosting service;and program instructions for encoding the graphical user interfaceelements for onward transmission to the remote client device forrendering, wherein the encoded graphical user interface elements aredecoded at the remote client device and rendered on a display of theremote client device, the graphical user interface elements map to atleast some controls of a physical game controller, and wherein inputsprovided at each of the graphical user interface elements rendered atthe remote client device are interpreted by the processor of the hostingservice to control a state of the online application.